[comment {-*- flibs -*- doctools manpage}]
[manpage_begin flibs/ftnunit n 1.2]
[copyright {2009 Arjen Markus <arjenmarkus@sourceforge.net>}]
[moddesc flibs]
[titledesc {Unit testing}]

[description]

[emph JUnit] is a well-known facility for defining and running unit
tests in Java programs. The [emph ftnunit] framework was inspired by
that facility. It is not as good-looking as JUnit, by no means:
[list_begin bullet]
[bullet]
It has a simple graphical user-interface that does not group the tests.

[bullet]
As Fortran does not allow introspection, the test routines can not
be detected automatically, instead as a programmer you need to set up a
high-level routine yourself that collects all the unit tests.

[bullet]
A runtime error, like division by zero, may lead to a termination of
the program. There is no (portable) way to catch these. Instead, the
framework relies on a batch file or shell script to repeatedly start the
program until all tests are run.

[list_end]

Despite these limitations, [emph ftnunit] can be a great help:

[list_begin bullet]
[bullet]
The Tcl program [emph gentabletest.tcl] generates a complete test program based
on a simple input file (see [sectref "GENERATING TESTS FROM A TABLE"]).

[bullet]
The code to test the various components (subroutines, functions, tasks
consisting of several program units) can be combined with the program
itself, without interfering with the ordinary code.
[nl]
This is achieved by defining a single routine (test_all, say) that runs
all the unit tests and that is called via the provided routine
[emph runtests]:

[example {
program myprog
    ...
!
! The routine runtests will check if unit tests are requested
! If not, it will return immediately. This way we make sure
! the unit tests remain part of the program.
!
! The routine test_all runs all unit tests
! (see the dataproc_testing module)
!
    call runtests( test_all )
!
! Ordinary processing
!
    ...

end program
}]

The routine runtests checks if there is a file "ftnunit.run". If there is
such a file, it will run the given subroutine. Otherwise it will return
and the rest of the program is executed.

[bullet]
Because the test code is incorporated in the program itself, it is less
likely that they evolve independently: changes in the argument lists of
the subroutines and functions may lead to compile errors in the test
code.

[bullet]
There is no need to set up a whole new program for testing portions of
the program.

[list_end]

The source file "test_ftnunit.f90" illustrates how to use the [emph ftnunit]
framework:

[list_begin bullet]
[bullet]
The main program calls the routine "runtests" and passes it the argument
"test_all", a routine defined in a module called "dataproc_testing".

[bullet]
The routine "test_all" consists of nothing but calls to the generic
routine "test":

[example {
subroutine test_all

    call test( test_no_file, "Read non-existent file" )
    call test( test_empty_file, "Read an empty file" )
    call test( test_invalid_file, "Read an invalid file" )
    call test( test_ordinary_file, "Read an ordinary file" )

end subroutine test_all
}]

[bullet]
The module includes a source file "ftnunit_test.f90". This is a remnant
of a previous version. Please ignore this.

[bullet]
The generic routine "test" checks whether a particular unit
test needs to be run (via the test number) and then runs the subroutine
that was passed as one of its arguments. One such routine looks like
this:

[example {
subroutine test_no_file

    integer :: nodata
    real    :: vmean, vmin, vmax

    call ftnunit_remove_file( 'no_such_file' )
    call write_name( 'no_such_file' )

    call open_files
    call process_data( nodata, vmean, vmax, vmin )

    call assert_true( nodata == 0, "No data read" )

end subroutine test_no_file
}]

The assertion is used to check that the result is as expected.

[bullet]
The program contains some deliberate errors and the resulting
log file looks like this:

[example {
Test: Read non-existent file
Test: Read an empty file
Test: Read an invalid file
forrtl: severe (59): list-directed I/O syntax error, unit 11, file c:\arjen\flibs\tests\ftnunit\invalid_file

Image              PC        Routine            Line        Source
test_ftnunit.exe     004151B9  Unknown               Unknown  Unknown
test_ftnunit.exe     00415017  Unknown               Unknown  Unknown
test_ftnunit.exe     004141F4  Unknown               Unknown  Unknown
test_ftnunit.exe     00414629  Unknown               Unknown  Unknown
test_ftnunit.exe     00409C05  Unknown               Unknown  Unknown
test_ftnunit.exe     004095FB  Unknown               Unknown  Unknown
test_ftnunit.exe     0040144B  Unknown               Unknown  Unknown
test_ftnunit.exe     00401FE9  Unknown               Unknown  Unknown
test_ftnunit.exe     00401A2C  Unknown               Unknown  Unknown
test_ftnunit.exe     00401BB3  Unknown               Unknown  Unknown
test_ftnunit.exe     0040294A  Unknown               Unknown  Unknown
test_ftnunit.exe     0040232E  Unknown               Unknown  Unknown
test_ftnunit.exe     0044A1E9  Unknown               Unknown  Unknown
test_ftnunit.exe     00433519  Unknown               Unknown  Unknown
kernel32.dll       7C816D4F  Unknown               Unknown  Unknown

Incrementally linked image--PC correlation disabled.
Test: Read an ordinary file
Number of failed assertions:                    0
Number of runs needed to complete the tests:    3
}]

[list_end]

The program is run via one of the following files:

[list_begin definitions]

[call runtests.bat]
A batch file for use under MS Windows
[call runtests.sh]
A Bourne shell script for use under UNIX/Linux or similar systems, like
Cygwin or Mingw.
[call runtests.tcl]
A Tcl program that presents a simple graphical user-interface
[call ftnunitgui.tcl]
A Tcl-based graphical user-interface that allows you to run tests one at a time

[list_end]


[section OUTPUT]
When you run a program in test mode via the batch file/shell script,
it produces two output files:

[list_begin bullet]

[bullet]
[emph "runtests.log"] contains all the output that was written to the screen.
[bullet]
[emph "ftnunit.html"] is an HTML file with a summary of the tests and a colour
encoding to indicate the test cases that succeeded or failed.

[list_end]


[section ROUTINES]
The module [emph ftnunit] contains the following subroutines and functions:

[list_begin definitions]

[call [cmd "call runtests( testproc )"]]
Routine to start the unit tests. It checks if the file "ftnunit.run"
exists. If so, it will call the subroutine [emph testproc] that was
passed. Otherwise it will simply return, so that the ordinary program
execution may continue.
[nl]
If the subroutine testproc returns, the program stops, unless you have
called the subroutine [emph runtests_init] before [emph runtests].

[list_begin arg]
[arg_def "subroutine" testproc]
Subroutine that calls the individual test routines. It takes no
arguments. It wil generally exist of a series of calls to the
routine [emph test] - see below.
[list_end]
[nl]

[call [cmd "call runtests_init"]]
Routine to initialise the ftnunit system, so that you call [emph runtests]
more than once. To complete the tests, call [emph runtests_final], as
this will print the final statistics and stop the program.
[nl]

[call [cmd "call runtests_final( stop )"]]
Routine to finalise the ftnunit system: it will print the final statistics
and stop the program, but only if the file "ftnunit.run" is present.

[list_begin arg]
[arg_def "logical, optional, intent(in)" stop]
If present and set to true, the routine will not stop the program.
Instead it returns and the program continues working.
[list_end]
[nl]

[call [cmd "call test( proc, text, ignore )"]]
Routine to run the individual unit test routine (emph proc). It decides
if the test has not run yet and if so, the test routine is called.
Otherwise it is skipped.
[nl]
[emph test] takes care of all administrative details.

[list_begin arg]
[arg_def "subroutine" proc]
Subroutine that implements an individual unit test. It takes no
arguments. Within each such subroutine the complete unit test is run.

[arg_def "character(len=*), intent(in)" text]
Text describing the particular unit test. It is printed in the log
file.

[arg_def "logical, optional, intent(in)" ignore]
If present and set to true, the test routine will not be actually run.
Instead it is shown as "ignored". This feature is useful if the code to
be tested is not yet ready for testing.
file.
[list_end]
[nl]


[call [cmd "call assert_true( cond, text )"]]
Routine to check that a condition is true. If not, a message is printed
in the log file and the number of failures is increased.

[list_begin arg]
[arg_def "logical" cond]
The condition to be checked

[arg_def "character(len=*), intent(in)" text]
Text describing the condition
[list_end]
[nl]


[call [cmd "call assert_false( cond, text )"]]
Routine to check that a condition is false. If not, a message is
printed in the log file and the number of failures is increased.

[list_begin arg]
[arg_def "logical" cond]
The condition to be checked

[arg_def "character(len=*), intent(in)" text]
Text describing the condition
[list_end]
[nl]


[call [cmd "call assert_equal( value1, value2, text )"]]
Routine to check that two logicals, two strings, or two integers are equal
or if two one-dimensional integer, logical or string arrays are equal. If not, a message is
printed, along with the values that were different.

[list_begin arg]
[arg_def "<type>" value1]
The first integer/logical/string value

[arg_def "<type>" value2]
The second integer/logical/string value

[arg_def "character(len=*), intent(in)" text]
Text describing the condition
[list_end]

Or:

[list_begin arg]
[arg_def "integer \[, dimension(:)\]" value1]
The first integer value or array

[arg_def "integer \[, dimension(:)\]" value2]
The second integer value or array

[arg_def "character(len=*), intent(in)" text]
Text describing the condition
[list_end]
[nl]


[call [cmd "call assert_comparable( value1, value2, margin, text )"]]
Routine to check that two reals are almost equal or if two one-dimensional
real arrays (single or double precision) are almost equal. If not, a message
is printed, along with the values that were different.
[nl]
The margin is taken as a relative tolerance. Two values are
considered almost equal if:
[example {    abs( v1 - v2 ) < margin * (abs(v1)+abs(v2)) / 2 }]

[list_begin arg]
[arg_def "real \[, dimension(:)\]" value1]
The first real value or array

[arg_def "real \[, dimension(:)\]" value2]
The second real value or array

[arg_def "character(len=*), intent(in)" text]
Text describing the condition
[list_end]
[nl]


[call [cmd "call assert_inbetween( value, vmin, vmax, text )"]]
Routine to check that a (single or double precision) real value lies between
two given bounds. This establishes an absolute range for the values, rather than
a relative.

[list_begin arg]
[arg_def "real" value]
The real value to be tested

[arg_def "real" vmin]
The minimum value that is allowed

[arg_def "real" vmax]
The maximum value that is allowed
[list_end]
[nl]


[call [cmd "call assert_files_comparable( filename1, filename2, margin, text )"]]
Routine to check that two files are equal or almost equal. The files are scanned
line by line and item by item. If an item can be interpreted as a number, then
the comparison is done using a margin, otherwise the corresponding items are
considered strings and should be exactly equal.
[nl]
A report is produced of all differing lines.
[nl]
The margin is taken as a relative tolerance. Two numerical values are
considered almost equal if:
[example {
    abs( v1 - v2 ) < margin * (abs(v1)+abs(v2)) / 2 }]

[list_begin arg]
[arg_def "character(len=*)" filename1]
The name of the first file to read

[arg_def "character(len=*)" filename2]
The name of the second file to read

[arg_def "character(len=*), intent(in)" text]
Text describing the condition
[list_end]
[nl]


[call [cmd "exists = ftnunit_file_exists( filename )"]]
Logical function to check that a particular file exists

[list_begin arg]
[arg_def "character(len=*), intent(in)" filename]
Name of the file to be checked
[list_end]
[nl]


[call [cmd "call ftnunit_get_lun( lun )"]]
Subroutine to get a free LU-number

[list_begin arg]
[arg_def "integer, intent(out)" lun]
Next free LU-number
[list_end]
[nl]


[call [cmd "call ftnunit_remove_file( filename )"]]
Subroutine to remove (delete) a file

[list_begin arg]
[arg_def "character(len=*), intent(in)" filename]
Name of the file to be removed
[list_end]
[nl]


[call [cmd "call ftnunit_make_empty_file( filename )"]]
Subroutine to make a new, empty file

[list_begin arg]
[arg_def "character(len=*), intent(in)" filename]
Name of the file to be created
[list_end]

[list_end]




[section "CUSTOMISATION"]
The module [emph ftnunit_hooks] can be used to customise the output
of [emph ftnunit] to a certain extent. It provides four routines
that you can easily adapt to your needs:

[list_begin definitions]
[call cmd "call ftnunit_hook_test_start( text )"]]
Routine to signal the start of a test.

[list_begin arg]
[arg_def string text]
The descriptive text passed to the routine [emph test].
[list_end]

[call cmd "call ftnunit_hook_test_stop( text )"]]
Routine to signal the end of a test.

[list_begin arg]
[arg_def string text]
The descriptive text passed to the routine [emph test].
[list_end]

[call [cmd "call ftnunit_hook_test_failure( text, assert_text, failure_text )"]]
Routine to signal that an assertion has failed (and therefore the test).

[list_begin arg]
[arg_def string text]
The descriptive text passed to the routine [emph test].
[arg_def string assert_text]
Text describing the assertion
[arg_def string failure_text]
The text describing the reason of the failure
[list_end]

[call [cmd "call ftnunit_hook_test_complete"]]
Routine called after the tests have been run (for instance to start and
Internet browser to view the HTML report).

[list_end]

Note that the default implementation consists of empty routine.

[para]
As an example of customisation, under Windows, you could start the
default Internet browser showing the HTML file like this:

    subroutine ftnunit_hook_test_complete

        call system( "ftnunit.html" )

    end subroutine ftnunit_hook_test_complete

provided an association for files with extension "html" exists.


[section "GENERATING TESTS FROM A TABLE"]
The Tcl program "gentabletest.tcl" reads the test specifications from an
input file and generates a complete Fortran program. The ideas from
Bil Kleb's  "Toward Scientific Numerical Modeling"
[uri ftp://ftp.rta.nato.int/PubFullText/RTO/MP/RTO-MP-AVT-147/RTO-MP-AVT-147-P-17-Kleb.pdf]
were used for the set-up.

[para]
To do: provide a detailed description. For the moment: see [emph example.tbl], including below.

[example {
! Example of generating test code via a table
! -------------------------------------------
! The routine to be tested determines the minimum oxygen concentration
! in a river, based on the Streeter-Phelps model:
!
!    dBOD/dt = -k * BOD
!
!    dO2/dt = -k * BOD + ka * (O2sat-O2) / H
!
! where
!    BOD   - biological oxygen demand (mg O2/l)
!    O2    - oxygen concentration (mg O2/l)
!    O2sat - saturation concentration of oxygen (mg O2/l)
!    k     - decay rate of BOD (1/day)
!    ka    - reareation rate of oxygen (m/day)
!    H     - depth of the river
!
! We need boundary (initial) conditions for BOD and oxygen and
! the equations describe the concentrations of BOD and oxygen in a
! packet of water as it flows along the river.
!
! Note:
! It is a very simple model, it is not meant as a realistic
! representation.
!
! The routine simply continues the solution until a minimum is found.
! The results are: oxymin and time
!
!
! The keyword DECLARATIONS introduces the declarations we need for the
! complete generated code
!
DECLARATIONS
    use streeter_phelps
    real :: bod, oxy
    real :: k, ka, h, oxysat, dt, oxymin, time
!
! The keyword CODE introduces the code fragment required to run the
! routine or routines. The results and possible checking of error
! conditions are separated.
!
CODE
    call compute_min_oxygen( bod, oxy, k, ka, h, oxysat, dt, oxymin, time )
!
! The keyword RESULT indicates which arguments/variables hold the
! interesting results. Specify one name per line (you can not currently
! use array elements) and the allowed margin (taken as absolute, if
! followed by "%" as a percentage)
!
RESULT
    oxymin  0.001        ! Minimum oxygen concentration
    time    0.01%        ! Time the minimum is reached

!
! The keyword ERROR is used for a code fragment that checks if the
! routine has correctly found an error in the input (that is, some
! parameter value is out of range). The code is invoked when any of
! result variables in a table entry has the keyword ERROR instead of
! a proper value.
! Use the subroutine "error" to indicate the correctly reported error
! condition.
!
ERROR
    if ( time == -999.0 ) then
        call error
    endif
!
! The keyword RANGES specifies that the variables are to be taken
! from a uniform or a normal distribution. The generated program will
! simply select values at random and run the code with them. The report
! consists of the detailed output as well as a summary.
!
RANGES
    oxy   10.0   2.0  Uniform ! Name of the variable, the mean and the margin (uniform)
                              ! Normal: mean and standard deviation followed by Normal
                              ! Note: all parameters must be given!
!
! The keyword TABLE indicates the beginning of a table of input data and
! expected values. The first (non-comment) line contains the names of
! the variables as used in the code fragments and all others are the
! values expected.
!
! There are two special values:
! ? -     indicating an unknown value for result variables and a "do not
!         care" value for input variables
!         It is useful to generate a table that does contain the (computed)
!         results (see the file table.out) or to indicate situations
!         where one or more input variables are out of range and this
!         should lead to an error
! ERROR - indicating that the entry should cause the routine to be
!         tested to flag an error condition.
!
TABLE
dt   oxy       bod       oxysat    h         k         ka        oxymin    time
0.1  10        1         10        10        0.1       1.0       10.0      2.0
1.0  10        1         10        10        0.1       1.0       ?         ?
!
! This case is unacceptable: time step must be positive
0.0  ?         ?         ?         ?         ?         ?         ?         ERROR
1.0  0.        10        10        10        0.1       1.0       ?         ?
}]

[section TODO]
The following things are still left to do:
[list_begin bullet]
[bullet]
Proper inclusion of the routine [emph prolog] and [emph epilog]

[bullet]
Extension of the set of assertion routines

[list_end]

[section "RELATED WORK"]
There are at least two similar initiatives with regard to a unit testing
framework for Fortran:

[list_begin bullet]
[bullet]
[uri {http://nasarb.rubyforge.org} {Funit (implemented in Fortran and
Ruby)}] by Bil Kleb and others

[bullet]
[uri {http://www.sourceforge.net/projects/pfunit} {A framework
implemented in Fortran}] by Brice Womack and Tom Clune

[bullet]
[uri {http://www.sourceforge.net/projects/fortranxunit} {FRUIT
(implemented in Fortran and Ruby)}] by Andrew Chen

[list_end]

(Note: To avoid confusion, I have renamed my original module "funit" to
[emph ftnunit])

[manpage_end]
